Image Acquisition and Display System and Method Using Information Derived from an Area of Interest in a Video Image Implementing System Synchronized Brightness Control and Use of Metadata

ABSTRACT

Method and apparatus for preparing and displaying images uses image characteristics of an area of interest of images to optimize or otherwise determine image characteristics of the entire image to be displayed. The area of interest information may be used in computing a System Synchronized Backlight Control (“SSBC”) profile tor the area of interest, and the SSBC profile may be used to adjust the image signal to provide an image in which the image characteristics, e.g.. are optimized for the image at the area of interest. The image characteristics and/or SSBC profile information may be provided as metadata included in a video signal data stream. The area of interest may be selected where the image is obtained, where the image is shown or elsewhere, e.g., in an image storage device, image player, etc.

CROSS-REFERENCE TO RELATED APPLICATIONS

The instant APPLICATION is a continuation of U.S. patent applicationSer. No. 12/525,098, filed May 21, 2010, now allowed, which is theNational Stage of International Application No. PCT/US08/52521 filedJan. 30, 2008, now published as WO 2008/095037, publication date Aug. 1,2008, which claims benefit of Provisional Application No. 60/945,667filed Jun. 22, 2007 and Provisional Application No. 60/887,346 filedJan. 30, 2007. All of the above-mentioned patent applications areincorporated herein by reference in their entirety as if fully set forthherein.

TECHNICAL FIELD

The invention relates generally, as indicated to image acquisition anddisplay systems and methods utilizing information from an area ofinterest and, more particularly, to an image acquisition and displaysystem and method using metadata derived from an area of interest in avideo Image to implement system synchronized brightness control

BACKGROUND

One type of display system, for example, a video display system or otherdisplay system, configuration consists of an illumination source(sometimes referred to as light source) and a pixilated, passivedisplay. The image is created by modulating the intensity of lightemitted from the illumination source simultaneous with transmissionthrough the display on a pixel by pixel basis. An example of such adisplay system is that based on the Liquid Crystal Display (sometimesreferred to as LCD). System Synchronized Brightness Control (sometimesreferred to as SSBC) (as well as variants on this approach) is a meansof simultaneously adjusting the backlight brightness and the gray levelsof the LCD pixels in real time based on the content of the input image.As one example using an SSBC approach in connection with preparing avideo signal for displaying on a display, the video signal may beadjusted, e.g., to stretch gray scale, to alter the intensity of thelight illuminating the display, etc. A purpose for these adjustments isto increase the inter frame contrast ratio, called the dynamic contrast,and lo enhance the details and the visibility of details as well as toreduce contouring in bright or dark areas of the image. Examples of SSBCapparatus and methods are disclosed, for example, in U.S. Pat. No.5,717,422, issued Feb. 10, 1998, U.S. Pat. No. 6,816,141 B1 issued Nov.9, 2004, and copending U.S. patent application Ser. No. 10/983,403,filed Nov. 8, 2004, the entire disclosures of which are incorporated bythis reference.

SUMMARY

Briefly, according to an aspect of the invention, an area of interest inan image is defined, and information pertaining to such area of interestis used to determine characteristics of the image for display.

According to another aspect, the area of interest in an image is usedlargely or exclusively for SSBC calculation, and the SSBC adjustmentsmay be implemented at the display or at the display end of a displaysystem. The definition of area of interest may be made through creativeinput of people or automatically through the use of an algorithm.

According to another aspect, the video imagery and the SSBC adjustmentsare encoded in the video stream,

According to another aspect, the video imagery and the SSBC adjustmentsare encoded in the video stream as metadata.

According to another aspect, the video imagery and the SSBC adjustmentsare encoded in the video stream as metadata separate from the videoimagery.

According to another aspect, the video imagery and the definition of thearea of interest, e.g., as determined by creative input of people, byautomatic implementation of an algorithm or by some other means, aretransmitted in the video stream, and the SSBC calculation is performedat the display or display end of a display system based largely or evenexclusively on the content of the area of interest. The SSBC adjustmentsare implemented at the display end.

According to another aspect, the rules for identifying an area ofinterest are included as an algorithm in the hardware at the display ordisplay end of a display system, and the SSBC calculations are performedat the display or display end based largely or even exclusively on thecontent of the area of interest; and the SSBC adjustments areimplemented at the display or display end.

According to another aspect, the information pertaining to an area ofinterest is used to calculate an SSBC “profile” of the image at thesource of the image. The source of the image is at the place and/orequipment where the image is obtained or initially is created or isadjusted, as contrasted to the place and/or equipment, such as adisplay, projector or the like, where or by which the image is presentedfor viewing, e.g., by a display, projector or the like.

According to an aspect of the invention, an area of interest in an imageis defined, and the SSBC “profile” of an image frame based on this areais calculated at the source of the image.

According to another aspect, the SSBC “profile” information calculatedat the source of the image is transmitted as metadata encoded into theinput image.

According to another aspect, a receiving device decodes and uses themetadata as the basis of means to implement the desired SSBCadjustments.

According to another aspect a receiving device, for example, a displaysystem, decodes and uses the metadata as the basis of means to implementthe desired SSBC adjustments.

According to another aspect, a receiving device, for example, a playerdevice, decodes and uses the metadata to implement the desired SSBCadjustments in the signal(s) provided a display to present an image.

According to another aspect, the information pertaining to such area ofinterest of an image is provided as metadata with the video signal orother signal representing the image and such metadata is used as thebasis of means to implement desired SSBC adjustments.

According to another aspect, a receiving device, for example, a displaysystem, decodes and uses the metadata that represents informationpertaining to an area of interest of an image as the basis of means toimplement the desired SSBC adjustments.

According to another aspect, a receiving device, for example, a playerdevice, decodes and uses the metadata that represents informationpertaining to an area of interest of an image to implement the desiredSSBC adjustments in the signal(s) provided a display to present animage.

These and further features of the present invention will be apparentwith reference to the following description and attached drawings. Inthe description and drawings, particular embodiments of the inventionhave been disclosed in detail as being indicative of some of the ways inwhich the principles of the invention may be employed, but it isunderstood that the invention is not limited correspondingly in scope.Rather, the invention includes all changes, modifications andequivalents coming within the spirit and terms of the claims appendedthereto.

Features that are described and/or illustrated with respect to oneembodiment may he used in the same way or in a similar way in one ormore other embodiments and/or in combination with or instead of thefeatures of the other embodiments.

It should be emphasized that the term “comprises/comprising” when usedin this specification is taken to specify the presence of statedfeatures, integers, steps or components but does not preclude thepresence or addition of one or more other features, integers, steps,components or groups thereof

Many aspects of the invention can be better understood with reference tothe following drawings. The components in the drawings are notnecessarily to scale, emphasis instead being placed upon clearlyillustrating the principles of the present invention. Likewise, elementsand features depicted in one drawing or embodiment of the invention maybe combined with elements and features depicted in one or moreadditional drawings or embodiments. Moreover, in the drawings, likereference numerals designate corresponding parts throughout the severalviews.

BRIEF DESCRIPTION OF THE DRAWINGS

In the annexed drawings:

FIG. 1 is a schematic illustration of an image or scene with an area ofinterest;

FIG. 2 is schematic block diagram of an image acquisition and displaysystem;

FIG. 3 is a graphical representation of a histogram of an exemplarynormal image showing 256 possible shades of gray and pixel counts forthe respective shades of gray;

FIG. 4 is a graphical representation of a histogram of an exemplary lowcontrast image;

FIG. 5 is a graphical representation of a histogram of an exemplary dimor relatively dark image;

FIG. 6 is a graphical representation of a histogram of an exemplaryrelatively bright image;

FIG. 7 is a graphical representation of a histogram of an SSBC adjusteddim image;

FIG. 8 is a schematic flow chart illustrating a method in accordancewith an embodiment of the invention;

FIG. 9 is a schematic block diagram of a portion of an image acquisitionand display system in accordance with an embodiment of the inventionwherein metadata pertaining to characteristics of an area of interest ofan image (or the image itself) is provided the receiver device, e.g.,display, which does an SSBC computation and adjusts the light source andgray levels accordingly; and

FIG. 10 is a schematic block diagram of an image acquisition and displaysystem in accordance with an embodiment of the invention in which avideo player or the like does the SSBC computation and the displaysystem adjusts the light source and gray levels accordingly withoutadditional computation.

DESCRIPTION

Initially referring to FIG. 1, an image 10 is illustrated in arectangular form. The image may be the view as seen through a cameralens, a view as received by film in a film camera, a digital image assensed by a digital camera, etc. The image 10 as it is shown in FIG. 1may actually be a real life scene prior to being picked up or sensed bya camera or the like. An area of interest 11 is in the image 10. Forexample, if the image 10 portrays a person and other background, thearea of interest 11 may be the face of the person or the entire body ofthe person, an airplane in a cloudy sky, a flower in a garden, etc. Thearea of interest 11 may he, for example, the place in the image 10 wherea movie director or editor wants a viewer to focus attention in theoverall image. The area of interest may be the entire image 10, e.g.,the full screen image, or it may be a portion less than the entireimage.

Several exemplary embodiments of the invention are described in greaterdetail below. Area of interest information, e.g., image characteristicsof the area of interest, may be used to determine, e.g., to compute, anSSBC profile for the image. The area of interest may be selected inseveral ways, e.g., as are described herein, and at different respectiveparts of a system, e.g., where the image is obtained, where the image isprepared for displaying, e.g., at a CD, DVD, tape, etc. player, at thedisplay, etc., as are described herein. The SSBC profile may beprepared, e.g., computed, calculated, etc., at different respectiveparts of a system, e.g., where the image is obtained, where the Image isprepared for displaying, at the display, etc., as are described herein,information pertaining to area of interest and/or SSBC profile may beprovided in a video stream, e.g., as metadata, for use where the imageis prepared for displaying, at the display, etc.

As is described farther below, the area of interest 11 may be selectedby a person who is photographing the image 10 by making appropriateadjustments on the camera or other equipment used to pick up or to sensethe image so as to identify a specified part or all of the image as thearea of interest. Alternatively, the area of interest 11 may be selectedduring editing of the image 10 by a person using a computer or otherediting equipment; for example, the area of interest may be highlighted,bounded by boundary lines or shapes, or otherwise selected byappropriate editing software or the like. The area of interest. 11 maybe selected automatically by software that is programmed to select anarea of interest according to specified criteria, for example. Forconvenience the image 10 also may be referred to below as an inputimage, e.g., a video signal, computer graphics signal, or some otherrepresentation of the image 10. The input image may be provided by acamera, by animation, or by some other source, e.g., a storage medium,such as a CD, DVD, tape, solid state memory, etc.

Turning to FIG. 2, an image acquisition and display system 12 isillustrated schematically in block diagram form. For convenience thesystem 12 will be referred to as including an image source portion orimage source 13 and an Image output portion or receiving device 14. Theimage output portion or receiving device 14 (sometimes referred to belowas “receiving device” or as “output portion”) may be, for example, adisplay, such as, for example, a passive display or other display. Asexamples, the image source portion 13 may be that part of the displaysystem at which an image is obtained; the receiving device or outputportion maybe a display on which an image is shown and/or may be adevice that prepares an image to be shown on a display, such as, forexample, a CD, DVD, blue ray (sometimes referred to as “Blu-Ray” or thelike) or tape player, etc.

In an embodiment of the invention at the image source portion 13 an areaof interest 11 in the image 10 is defined, the SSBC profile of an imageframe Is calculated based on such area of interest, and the SSBCinformation is transmitted as metadata encoded into the input image. Inan embodiment of the invention the receiving device 14 decodes and usesthe metadata as the basis of implementing the desired SSBC adjustments.This approach may provide one or more advantages. For example, it mayensure proper calculation and encoding of the desired adjustments intothe input image. Also, it may reduce system cost by moving the burden ofSSBC calculation and encoding and the associated expense of the hardwarerequired to enable such calculation to the source, e.g., the imagesource portion 13, where it can be done once for each source image, thusnot requiring it to be done at each and every individual display.Further, it may facilitate providing image adjustment based on area ofinterest characteristics. A simple metadata decoding process in eachdisplay will assure the proper display of the image. The SSBC techniquecan be applied to displays that are made up of independent illuminationand image sources.

In accordance with embodiments of the invention SSBC data or informationand information or rules of selection with regard to area of interestmay he provided as metadata or may be otherwise provided. Examples thatmay use, but do not necessitate use of, metadata are, as follows:

The area of interest in an image may he used largely or exclusively forSSBC calculation, and the SSBC; adjustments may be implemented at thedisplay or at the display end (also referred to as the output portion)of a display system. The definition of area of interest may be madethrough creative input of people or automatically through the use of analgorithm.

The video imagery and the SSBC adjustments are encoded in the videostream.

The video imagery and the definition of the area of interest, e.g., asdetermined by creative input of people, by automatic implementation ofan algorithm or by some other means, may be transmitted in the videostream, and the SSBC calculation may be performed at the display ordisplay end of a display system based largely or even exclusively on thecontent of the area of interest. The SSBC adjustments are implemented atthe display end.

The rules for identifying an area of interest may be included as analgorithm in the hardware at the display or display end of a displaysystem, and the SSBC calculations may be performed at the display ordisplay end based largely or even exclusively on the content of the areaof interest; and the SSBC adjustments are implemented at the display ordisplay end.

As is described in an embodiment of the invention, the area of interestwithin an image and the image characteristics or feature(s) of the areaof interest may be used as the basis upon which SSBC adjustments arecomputed. The area of interest can be defined in several ways, severalexamples of which are, as follows:

The in-focus portion of the image: An image can be created having anarrow depth of field, in such an image, the area of interest is insharp focus while other portions of the image, e.g., those either closerto or further from the viewer, are out of focus.

Faces: If a face occupies a large percentage of an image, it is likelythe area of interest. Various face recognition software programs andalgorithms are available, for example, to recognize either that an imageor a portion of an image is a face or that not only is the image a face,but the face is a particular face belonging to a specific individual.

Still object moving against a moving background: As an example, considera close up image of a passenger in a moving vehicle. The passenger'simage is relatively stationary against the moving background of thescene outside the vehicle; the stationary object is the area ofinterest.

Moving object against a still background: Consider a horse and ridermoving across a field; the moving object is the area of interest.

Bright area in an otherwise poorly lit image: Consider a singer standingin a spotlight on an otherwise dark stage; the illuminated portion ofthe image is the area of interest.

Colorful area in an otherwise uncolorful image: Consider several brightor multi-colored balloons floating across an overcast sky; the colorfularea is the area of interest.

In another embodiment of the invention at the image source portion 13 anarea of interest 11 In the image 10 Is defined, and informationpertaining to characteristics of the area of interest, e.g., one or moreof contrast, brightness, colors, etc., is transmitted as metadataencoded into the signal representing the input image and is provided theoutput portion 14, e.g., a display, a player that provides signals to beshown on or projected by a display, etc., and at the output portion themetadata may be decoded and appropriate SSBC computation may be carriedout. For example, the signal representing the input signal may be avideo signal, and the metadata may be combined with the video signal.The computation carried out at the output portion 14 to obtain thedesired SSBC profile to provide SSBC adjustment need not be as complexand time consuming as would be the case if the entire SSBC effort werecarried out at the output portion, e.g., determining contrast,brightness and colors and also performing the computation.

In the two embodiments mentioned just above, the advantages of the SSBCfeatures may he effectively used in connection with displayed orprojected images or with images that are otherwise provided or used.Also, image adjustment conveniently may be carried out based oncharacteristics of an area of interest, which may enhance the overallimage and/or also may help to focus or to direct the attention of aviewer, for example, to the area of interest, e.g., as the image isdisplayed, projected or otherwise shown or provided to a person viewingthe image. Further, the area of interest may be selected by thephotographer, editor, etc, before the image, e.g., the video signal withthe metadata, is provided the output portion 14.

The image source portion 13 may include a camera 15 that picks up orsenses an image or may include some other source of images, e.g.,animated images or other image representations, e.g., computer graphicssignals or virtually any other source of images. The image sourceportion also includes a computing or calculating device 16, such as, forexample, a computer, e.g., a processor, a personal computer, or othercomputer, possibly including associated peripheral and/or internalparts, such as, for example, a memory 17, display and input/outputequipment 18, e.g., keyboard, display, mouse, trackball, and/orjoystick, etc. The computer 16, memory 17, and display and input/outputequipment 18 may be used to select, for example, manually (e.g., by anoperator of the equipment) or automatically, an area of interest 11 ofan input image 10, to compute an SSBC profile for the area of interest,and to provide the SSBC profile or information or to provide informationpertaining to characteristics of the area of interest (e.g., one or moreof contrast, brightness, colors, etc.) as metadata in a signal, e.g., avideo signal, that is to be provided the image output portion 14. Aperson of ordinary skill in the art may write appropriate computerprogram software, code or the like in an appropriate language to carryout the steps and functions disclosed herein. Various other parts of thesystem 12 also may include a processor or the like to carry out thevarious functions described herein, e.g., to determine area of interest,to determine image characteristics of the area of interest, and/or tocompute an SSBC profile based on Image characteristics of the area ofinterest, as are described elsewhere herein.

The content of the “captured image,” that is, the as recorded videoimage, for example, may be analyzed. As an example, the means used tomake this analysis are histograms of the image content in which the“GRAY SHADE” of the pixels is plotted against the “PIXEL COUNT” ornumber of pixels in the display having that gray shade value. In theexamples described with respect to FIGS. 3-7, the gray shades range fromshade 1 at the dark extreme to shade 256 at the bright extreme.

FIG. 3 is a histogram 33 in which the captured image is “normal” whichIn the context of the example presented herein means an image in whichthe pixel brightness extends more or less uniformly over the entirerange of available gray shades, in FIGS. 3-7 the shades of gray areshown along the X-axis (horizontal axis of the illustrated graphs orcharts) and the number of pixels having a given shade of gray is plottedon the Y-axis (vertical axis of the illustrated graphs or charts). FIG.4 is a histogram 34 of a low contrast image. Note that only a limitedportion of the available gray shades are used In the image. FIG. 5 is ahistogram 35 of a dim image. Note that the image is composed of only thelow numbered gray shades. FIG. 6 is a histogram 36 of a bright image.Note that the image is composed of only the high, numbered gray shades.The dim image and the bright image are similar to the low contrast imageIn that ail utilize only a small portion of the gray scale range thatthe display is capable of producing. One consequence of utilizing alimited range of gray shades in an image is that it may not be possibleto display brightness subtleties in the image. The result is thevisually undesirable effect called contouring.

Some types of video display systems are composed of two elements: asource of illumination and a separate, passive pixilated display. (Apassive display is one that modulates rather than emits light.) Anexample of this type display system is a backlit LCD as is commonlyfound in direct view LCD television, monitor and laptop computerproducts, as well as microdisplay (HTPS, LCOS, DLP) based projectiontelevisions. In a conventional implementation of such a display system,tire brightness of the illumination source is held constant and thevarious gray shades in the image are accomplished through a pixel bypixel modulation by the LCD.

SSBC is a process by which the content of the “captured image” isanalyzed in real time and the brightness of the illumination source andthe range of gray shades in the image are simultaneously adjusted Inreal time. For example, the content of an input video image may beanalyzed in real time and the brightness of the illumination source andthe range of gray shades in the image are simultaneously adjusted inreal time in such a way as to increase the inter frame contrast ratioand to better utilize the available shades of gray. One example of theprocess proceeds as follows:

-   -   The input image or the “captured image” is analyzed by creating        a histogram to determine the number of pixels at each gray scale        level for a frame of video. An algorithm can be used to        characterize the distribution of pixels and gray levels into        several classes. For example, the range of pixel gray shade        distribution and the average image brightness can be calculated        and used in a control algorithm. Various algorithms have been        developed and used for this purpose, e.g., by respective        television manufacturers that include SSBC features in their        televisions, and other algorithms may be developed in the future        for these purposes.    -   The histogram distribution of the input image or “captured        image” or the gray scale range can be stretched such that the        displayed image utilizes more of the available gray levels. The        stretching can he accomplished, for example, by application of a        non-linear gamma function or gamma type of function. Techniques        for adjusting gamma in display systems are blown. The actual        non-linear gamma function or a function similar to the type of        function that is used to apply a gamma characteristic or a gamma        adjustment in a display may depend on the characteristics of the        display and may be written accordingly by a person who has        ordinary skill in the art. Not only can the histogram        distribution or the gray scale range be stretched, but other        algorithms can interpolate and insert additional gray levels        into the stretched image. The additional gray levels tend to        reduce contouring.    -   A synchronous adjustment is made to the illumination source to        compensate for the brightness change that occurs because of the        histogram stretching. For example, the appropriate light        intensity may be provided the passive display to obtain the        desired brightness of the output Image that is viewed or        projected.

The result of the SSBC adjustments is the production of the “SSBCsourced, processed and encoded image” that is to be sent to the display.

As an example of this process, an illustration of the histogram 37 of aSSBC adjusted dim video image is presented in FIG. 7. For example, inthe dim image illustration of FIG. 5 the shades of gray used in theimage are in Range 1, which extends from about a shade 7 to about ashade 49. To do an SSBC adjustment of the dim video image to stretch, toexpand or to increase the number of shades of gray, or to maintain thesame number of shades of gray but to have them extend over a wider rangeof shades than those of the illustrated dim image, as is illustrated inFIG. 7, the shades of gray now extend over Range 2, e.g., from aboutshade 5 to about shade 250.

After SSBC adjustment, a wide portion of the available gray shades canbe displayed under all video image conditions. The resulting image thatmay be visible on the display can be called “displayed image”.

In FIG. 8 a flow chart 38 outlining the process by which SSBC istypically applied to a video signal is illustrated. At the head end 40(where content generation and recording occur) the image is analyzed andthe proper SSBC calculations are performed and encoded into the videosignal, image capture, editing, recording and compression and encodingsteps are shown in the block 40, for example. The steps at block 40 maybe carried out in the image source portion 13 (FIG. 1). The videosignal, with encoded SSBC information, is then stored or is transmittedas shown at block 41, e.g., via various broadcasting, satellite,internet, etc. transmission and/or via various storage means such as,tor example, CD, DVD, Blu-Ray, HD DVD, etc. Finally, at the receivingend 42, the video image with SSBC is implemented and the images aredisplayed. The steps at block 42, decoding of metadata, SSBC processingof video signal, SSBC adjustment of illumination and display may becarried out at the output portion 14 (FIG. 1), for example.

In prior display systems that used SSBC technique, SSBC analysis andcalculation have been performed at the location of the display and canbe called the in-display approach or in-display system. This approachnecessitates that all of the digital control circuitry cost be incurredin each and every display. In addition, it leaves the nature of the SSBCadjustment in the hands of the display designer and not the contentcreator, that is, in a sense in the hands of engineers and not artists.

The present invention may be used to help provide for the SSBCcalculations to be accomplished fully and properly for many differentvideo display systems. In the present invention the SSBC calculationsand implementation and decisions pertaining to them may be provided fromboth image creators and video engineers, for example, such videoengineers who design displays.

In prior in-display SSBC systems the procedure for determining theproper adjustment to the image brightness and the gray levels, sometimesreferred to as proper SSBC calculation, various manufacturers,organizations and individuals have developed, means of image analysiswith the result that many different algorithms have been developed. Thevarious approaches for in-display SSBC use different methods and haveimproved the image quality to differing extents, e.g., not the same fordifferent displays, etc.

There is a burden or cost associated with in-display SSBC adjustmentsbecause the video signal is analyzed and the SSBC calculation isaccomplished at the receiving end. That is, by circuitry co-located ator even within the video display. This approach means that appropriatecircuitry for input image analysis and application of algorithms needsto be included along with or in every video display system. 1 lieexpense of implementing SSBC is fully incurred in each and everyindividual video display system unit.

In accordance with an aspect of the invention, the video image may beprocessed at its source and the incoming video signal is encoded withthe information on the proper SSBC implementation. These aspects of theinvention tend to reduce the need for circuitry in each individual videodisplay to perform (at least) the SSBC calculations. As is describedfurther herein, the SSBC process may he accomplished in a way such thatthe “modified” video signal, e.g., modified according to SSBC to achieveSSBC image quality or enhancements, can still be properly displayed on aconventional (e.g., non SSBC enabled) video display.

According to an aspect of the invention, as is described herein, theproper SSBC calculation is generated, and information specifying theproper SSBC implementation is generated at the head end of the system,e.g., the image source portion 13 (FIG. 1), thus removing the need forcalculation circuitry at each and every receiver. These aspects may beaccomplished in such a way that the resulting modified video image canbe properly displayed on conventional (non SSBC enabled) displays. Oneexemplary approach to accomplish these aspects Is to base the SSBCadjustment on a visual “area of interest” within the image, as isdescribed elsewhere herein.

SSBC can be applied and the benefits of SSBC accomplished in direct viewand projection type LCD displays, The reason is that, in these typedisplay systems, the illumination source and the passive, pixilateddisplay are separate entities.

On the average, the field of view of each individual human eye in thehorizontal plane is about 150° (150 degrees) (e.g., about 60° to nose,and about 90° to side). The vertical field of view is about 120° (e.g.,about 60° up, and about 60° down). Note, however, that within the fieldof view, the attention of the human eye/brain system will, at any time,focus on a much smaller area, called the area of interest. This isusually the case even when the viewer is looking at a video display.That is, within the display, regardless of its size, the human eye/brainwill focus on a sub area of an image, regardless of the size of theimage. The size and location of the sub area can change with time but,none-the-less, the primary, most detailed information that the viewerperceives of the image usually is from the area of interest. That is,the viewer's impression of overall image quality is determined by thequality of the image within the area of interest.

According to an embodiment of the invention, in order to provide theviewer a high quality image, the quality of the image within the area ofinterest is optimized or at least is improved, for example, using SSBCmethods. The adjustment or quality of the image outside the area ofinterest is less important to the viewer's determination of imagequality than that of the image quality within the area of interest.

There are a variety of definitions commonly used for the term metadata.Several simple definitions are as follows:

-   -   Metadata is data about data.    -   Metadata is information about data.    -   Metadata is information about information.

An example of metadata and of its use and value is as follows: “12345”is data which, without additional context, is meaningless. Whenadditional information is provided (metadata) such as “this number is aZIP code” (postal code), one can understand that “12345” refers to theGeneral Electric plant in Schenectady, N.Y.

A more sophisticated definition of metadata is:

-   -   Metadata is optional, structured, encoded data that are publicly        available and that describe characteristics of information        bearing entities to aid in the identification, discovery,        assessment and management of the described entities.

Metadata is utilized in video technology. To elaborate on the role ofmetadata in video, consider MPEG-7, an ISO standard developed by theMoving Picture Experts Group. MPEG-7 can be described as the multimediastandard for the fixed and mobile web enabling integration of multipleparadigms. An alternative and somewhat supplementary description is thatMPEG-7 is a standard for describing the multimedia content data thatsupports some degree of interpretation of the information, meaning,which can be passed on to, or accessed by, a device or a computer code.

Further details on how metadata is embodied in the MPEG-7 standard areas follows: MPEG-7 Includes Visual Description Tools that consist ofbasic structures and Descriptors. They cover the following basic visualfeatures: color, texture, shape, motion, localization and facerecognition. MPEG-7 Multimedia Description Schemes (DSs) are metadatastructures for describing and annotating audio visual (AV) content. TheDSs provide a standardized way of describing in XML (a general, purposemarkup language that supports a wide variety of applications) theimportant concepts related to AV content description and contentmanagement in order to facilitate searching, indexing, filtering andaccess. The DSs are defined using the MPEG-7 Description DefinitionLanguage and are instantiated as documents or streams. The resultingdescriptions can be expressed in a textual form (e.g., human readablefor editing, searching, filtering) or compressed binary form (e.g., forstorage or transmission).

The MPEG-7 Descriptors are designed primarily to describe low levelaudio or visual features such as color, texture, motion, audio energyand so forth, as well as attributes of AV content such as location,time, quality and so forth. It is expected that most Descriptors for lowlevel features shall be extracted automatically in applications.

On the other hand, the MPEG-7 DSs are designed primarily to describehigher level AV features such as regions, segments, objects, events; andother immutable metadata related to creation and production, usage andso forth. The DSs produce more complex descriptions by integratingtogether multiple Descriptors and DSs, and by declaring relationshipsamong the description components. In MPEG-7, the DSs are categorized aspertaining to the multimedia, audio or visual domain. Typically, themultimedia DSs describe content, consisting of a combination of audio,visual data and possibly textual data, whereas, the audio or visual DSsrefer specifically to features unique to the audio or visual domain,respectively.

To restate these comments on metadata in less technical terms and tofocus them on video: metadata is additional information that istransmitted along with a video signal The data occupies otherwise unused“space” in the video signal. Tins additional information can he ignoredby the receiving system with no effect on the image. On the other hand,if the receiving system is properly enabled, the metadata can beinterpreted and so provide further information about the image. Inprinciple, this additional information can he used at the location ofthe display to adjust the qualities of the video image.

An image acquisition and display system, tor example, as is illustratedat 12 in FIG. 2, may be used to carry out the following process ormethod:

-   -   At the source, e.g., the image source portion 13, define a        visual area of interest 11 in each and every video frame 10. The        source can be the point of image creation, image editing, image        recording or image transmission, for example.    -   At the source, circuitry, e.g., computer 16 and associated        peripherals and software, calculates the adjustments to the        backlight brightness and the image pixel data including gray        levels required to optimize image quality within the area of        interest. As is described herein, to optimize image quality        means at least to improve the image quality, for example by        using the SSBC approach described herein, e.g.. to increase or        to stretch the number of gray levels at which linage data is        shown by the display and to make corresponding adjustment of the        illumination level (brightness or intensity) of the light source        that illuminates the passive display.    -   At the source, encode this information as metadata.    -   Record (CD, DVD or store for internet distribution), e.g., at 51        (FIG. 2) or transmit, e.g., as is shown at 52, the metadata        along with the video signal. The video signal and metadata may        be provided at the output 53 of the image source portion 13, for        example.    -   At the display 54, circuitry decodes the backlight brightness        and the pixel information including the gray levels instructed        by the metadata.    -   At the display, circuitry, which receives transmitted signals        from 52 or signals from a CD/DVD/tape or other medium player 55,        adjusts the backlight 56 brightness and the LCD gray levels,        e.g., to be portrayed by the passive display 57, as instructed.

Consider what is accomplished by utilizing the disclosed procedure:

-   -   Since it is performed at and by the source 13, the SSBC        calculation can be optimized by the image creator, editor,        encoder or broadcaster such that the displayed image by 14 will        have the intended appearance.    -   Note that SSBC adjustments made at the source 13 can be chosen        to be consistent with the artistic intentions of the image        creator.    -   Because the system 12 optimizes or at least improves the image        quality within the area of interest 11, the viewer perceives an        overall improvement in image quality. Optimizing or improving        image quality may include, for example, expanding or stretching        gray scale, enhancing contrast and/or brightness of the image,        enhancing detail provided in and/or seen in the image, setting        maximum and/or minimum levels (e.g., brightness, contrast, etc.        of the image), providing contrast weighting, etc.    -   Since functions described above are done at the source 13, the        burden of SSBC calculation is incurred just once. This removes        the need for calculation circuitry at every receiver thus        reducing overall system cost.    -   Note that display systems that are not SSBC enabled will ignore        the metadata. The image displayed on such systems will be        unaffected by the presence of the metadata.

Different levels of metadata instructions can be encoded to improve thedisplayed image on a wide range of displays, from high definition TVs tomobile cell phone displays.

As is illustrated in FIG. 9, another embodiment of the invention has themetadata contain information related to characteristics of the area ofinterest including, for example, its contrast, brightness, colors andthe like. Such “characteristics metadata” may be provided by the sourceportion 13. In FIG. 9 the characteristics metadata is provided at 16 afrom the source portion 13 (FIG. 2); the video signal with thecharacteristics metadata is shown at block 53′, which is provided as aninput to the output portion 14 of the system 12 a. The output portion 14includes a player, e.g., CD, DVD, Tape, Blu-Ray, etc., or display 55′,light source(s) 56 and a passive display 57. The player or displaysystem 55′ is associated with the light source(s) and display andincludes a computer 60 and memory 61 to compute the appropriate lightintensity for illuminating the display and revised, e.g., stretched,shades of gray at the output portion 14, thereby to provide an imagethat is adjusted according to the area of interest information receivedat 16 a. in this embodiment, the SSBC enabled receiver 14 would stillhave the capability to calculate the required adjustments to thebrightness of the backlight 56 and the gray shades for the passivedisplay 57, Advantages of this embodiment include the ability of eachindividual model and make of receiver to implement adjustmentsappropriate to its unique display characteristics. Gamma adjustments aremade in respective displays whereby the gamma characteristic of thedisplay is taken into consideration as incoming signals are processedand shown on a display; and the present invention may similarly takeinto account the unique characteristics of a display as an SSBC or likemethod is implemented based on received metadata and/or based on area ofinterest data. Thus, this embodiment still assures that the correctionsare properly identified and optimally implemented. This embodiment maynot lead to such reduction in cost as may be achieved in the embodimentof FIG. 1. for example, because of the need to maintain the ability tocalculate the changes at each receiver; but the time to make thecalculations may be reduced because the image characteristics areprovided by the source 13 and/or because in some instances thecalculations may only have to be carried out for the area of interest 11that is smaller than the entire image 10.

In FIG. 10 another embodiment of the invention Illustrated includes theSSBC calculation capability in an advanced video player such as a DVD,Blu-ray or HD DVD player. The video signal may be recorded at recorder51 may include image characteristics metadata, e.g., of an area ofinterest, which is shown at block 53′. in this embodiment, the videoplayer 55″ would not only play the DVD, etc., but would perform the SSBCimage analysis and add appropriate metadata to the video stream. TheSSBC enabled display would properly decode the metadata information andpresent an enhanced image.

Several additional embodiments of the invention are described below.

In an embodiment an example of metadata encoding includes providing fullinstructions; in a sense the display is a “dumb” display in that It doesnot have to do any calculations. Full instructions are provided to thedumb display. In this embodiment, the SSBC calculations are performed atthe head end 13 (FIG. 1) or 40 (FIG. 8). The calculations can be basedon an image area of interest 11 or some other criteria. The calculationscan utilize Input from people or be accomplished automatically by asoftware algorithm. The video stream along with the metadata are storedor transmitted. Upon receipt at the display end 14, the instructions aredirectly implemented without further processing or interpretation. Thiscan be thought of as a SSBC enabled “dumb” display.

Several advantages of this approach include;

-   -   Reducing the cost of the hardware and software at the display        end.    -   “Getting it right” since the encoding is done at the content        source by the image creators.

The invention may be used with a display that is responsive to SSBCprofile information or with displays that are not able to respond toSSBC profile information. Therefore, using the present invention, evenif the area of interest and/or SSBC profile information are provided thedisplay as metadata at an unused portion of a video signal data stream,such metadata will not affect performance of the display. Thus, in thissense, the invention is backwards compatible being useful with displaysystems capable to make adjustments in the image characteristics orimage signal based on area of Interest image characteristics or SSBCprofile arid with display systems that are not so capable; in the lattercase the area of interest and/or SSBC information may be ignored by thedisplay or will not be seen or considered by the display.

In another embodiment Instructions are provided in a look up table inthe display. The look up table may be specific to a display model. Inthis embodiment, the SSBC calculations are performed at. the head end 13or 40. The calculations can be based on an image area of interest 11 orsome other criteria. The calculations can utilize input from people orbe accomplished automatically by a software algorithm. The video streamalong with the metadata are stored or transmitted. Upon receipt at thedisplay end 14, the required adjustments are referred to a look up tableunique to the display make and model. In this way the SSBC adjustmentsthat are implemented at the display and the displayed image, includingthe light source intensity and the passive display gray levels areadjusted based on values in the “look up” table or found in the look uptable to tend to enhance or to optimize the Image for the display makeand model.

Several advantages of this approach include:

-   -   The same system of encoding applies across the entire spectrum        of display ends 14 (also referred to above as output portion or        receiving device).    -   The system is “hierarchical. That is, some sophisticated display        ends may use the entire range of look up table information        encoded in the metadata but other, less complex display ends may        use only a sub set.

In another embodiment metadata is calculated by a smart display. In thisembodiment the definition of an area of interest 11 or some othercriteria are stored or transmitted as metadata along with the videostream. Upon receipt at the display end 14, the SSBC adjustments arecalculated by a software algorithm in the media processor (in the caseof a television, for example) or in the module controller (in the caseof a LCD, for example). The SSBC adjustments are then implemented on thedisplay either directly or, through a look up table, in a manner uniqueto the specific make and model of the display.

Regarding location for generating metadata, in various embodiments Inwhich the SSBC calculation is performed at the display end 14 (eitherbased on an area of interest 11 or some other criteria) the function canbe performed in a number of device types, e.g., at a player or otherdevice that provides images to the display. This is useful because ofthe large body of “legacy” video (video not originally created with SSBCincluded) that will need to be processed before being displayed on anSSBC enabled display. Examples of devices in which the SSBC calculationand the encoding function can be performed include the following:

-   -   DVD player    -   Blue-Ray player    -   HDDVD player    -   A computer    -   A cable or satellite box    -   A digital tuner of set top box    -   Video game consol

It is noted that the size of the area of interest 11 is flexible. In theextreme case, it occupies the entire image and thus reduces to the mostelementary procedure used for SSBC adjustment calculation.

Also, it will be appreciated that the features of the inventiondescribed herein apply to still images as well as to multiple Images,e.g., movies, videos, etc.

Metadata reduces calculation time and lag at the display end 14.Consider, for example, a case in which the image pixel count is high andthe algorithm for calculation of SSBC adjustment is complex. If the SSBCcalculations were implemented at. the display end 14 it is conceivablethat the ability of the display system to process the image may not heable to keep up with the demands of the video rate. In such a case,visual artifacts may be introduced into the Image. This points to apotential advantage of the metadata approach. Since the calculation isperformed at the head end 14 or 40 and transmitted along with the videosignal, the burden of calculation Is removed from the display end 14. Inthis way, the ability of the system 12, for example, to display SSBCenabled video at full video rate is assured.

There are several different and potentially quite distinct types ofvideo materials available including:

-   -   Movies—action, drama, romantic    -   Sports—indoor, outdoor    -   Video games    -   Animation    -   Broadcast TV—talk shows, sitcoms

An example of a method and system used in embodiments of the Inventionto enhance the ability of the invention to properly display thesedifferent types of video materials having alternate SSBC look up tables.The table that is selected is determined by user choice or a contentcreator specification that is included in the metadata.

Although the invention is described generally without specific referenceto color displays and video signals representing color images, black andwhite displays and video signals representing black and white images, orother specific type of displays or video signal, it will be appreciatedthat the features of the invention may be used with color displays,black and white displays, and/or other types of displays and withvarious types of signals, e.g., video signals, MPEG format signals, etc.

It will be apparent to a person having ordinary skill in the art ofcomputer programming, and specifically in application programming fordisplay systems and/or other electrical and/or electronic devices wouldbe able to prepare appropriate computer program software or code tooperate and carry out the various features and logical functionsdescribed herein, e.g., for use in the computer 16 and other parts ofthe system described herein. Accordingly, details as to specificprogramming code have been left out for the sake of brevity.

Although the invention has been shown and described with respect tocertain preferred embodiments, it is understood that equivalents andmodifications will occur to others skilled in the art upon the readingand understanding of the specification. The present invention includesall such equivalents and modifications, and is limited only by the scopeof the following claims.

It will be appreciated that portions of the present invention can beimplemented in hardware, software, firmware, or a combination thereof.In the described embodiments), a number of the steps or methods may beimplemented in software or firmware that is stored in a memory and thatis executed by a suitable instruction execution system. If implementedin hardware, for example, as in an alternative embodiment,implementation may be with any or a combination of the followingtechnologies, which are all well known in the art: discreet logiccircuit(s) having logic gates for implementing logic functions upon datasignals, application specific integrated circuit(s) (ASIC) havingappropriate combinational logic gates, programmable gate array(s) (PGA),field programmable gate array(s) (FPGA), etc.

Any process or method descriptions or blocks in How charts may beunderstood as representing modules, segments, or portions of code whichinclude one or more executable instructions for implementing specificlogical functions or steps in the process, and alternate implementationsare included within the scope of the preferred embodiment of the presentinvention in which functions may be executed out of order from thatshown or discussed, including substantially concurrently or in reverseorder, depending on the functionality involved, as would be understoodby those reasonably skilled in the art of the present invention.

The logic and/or steps represented in the How diagrams of the drawings,which, for example, may be considered an ordered listing of executableinstructions for implementing logical functions, can be embodied in anycomputer-readable medium for use by or in connection with an instructionexecution system, apparatus, or device, such as a computer-based system,processor-containing system, or other system that can fetch theinstructions from the instruction execution system, apparatus, or deviceand execute the instructions. In the context of this document, a“computer-readable medium” can be any means that can contain, store,communicate, propagate, or transport the program for use by or inconnection with the instruction execution system, apparatus, or device.The computer readable medium can be, for example but not limited to, anelectronic, magnetic, optical, electromagnetic, infrared, orsemiconductor system, apparatus, device, or propagation medium. Morespecific examples (a nonexhaustive list) of the computer-readable mediumwould include the following: an electrical connection (electronic)having one or more wires, a portable computer diskette (magnetic), arandom access memory (RAM) (electronic), a read-only memory (ROM)(electronic), an erasable programmable read-only memory (EPROM or Flashmemory) (electronic), an optical fiber (optical), and a portable compactdisc read-only memory (CDROM) (optical). Note that the computer-readablemedium could even be paper or another suitable medium upon which theprogram is printed, as the program can be electronically captured, viafor instance optical scanning of the paper or other medium, thencompiled, interpreted or otherwise processed in a suitable manner ifnecessary, and then stored in a computer memory.

The above description and accompanying drawings depict the variousfeatures of the Invention. It will be appreciated that the appropriatecomputer code could be prepared by a person who has ordinary skill inthe art to carry out the various steps and procedures described aboveand illustrated in the drawings. It also will be appreciated that thevarious terminals, computers, servers, networks and the like describedabove may be virtually any type and that the computer code may beprepared to carry out the invention using such apparatus in accordancewith the disclosure hereof.

Specific embodiments of an invention are disclosed herein. One ofordinary skill in the art. will readily recognize that the invention mayhave other applications in other environments. In fact, many embodimentsand implementations are possible. The following claims are in no wayintended to limit the scope of the present invention to the specificembodiments described above. In addition, any recitation of “means for”is intended to evoke a means-plus-function reading of an element mid aclaim, whereas, any elements that do not. specifically use therecitation “means for”, are not intended to be read asmeans-plus-function elements, even if the claim otherwise includes theword “means”.

Although the invention has been shown and described with respect to acertain preferred embodiment or embodiments, it is obvious thatequivalent alterations and modifications will occur to others skilled inthe art upon the reading and understanding of this specification and theannexed drawings. In particular regard to the various functionsperformed by the above described elements (components, assemblies,devices, compositions, etc.), the terms (including a reference to a“means”) used to describe such elements are intended to correspond,unless otherwise indicated, to any element which performs the specifiedfunction of the described element (i.e., that Is functionallyequivalent), even though not structurally equivalent to the disclosedstructure which performs the function in the herein illustratedexemplary embodiment or embodiments of the invention. In addition, whilea particular feature of the invention may have been described above withrespect to only one or more of several illustrated embodiments, suchfeature may be combined with one or more other features of the otherembodiments, as may be desired and advantageous for any given orparticular application.

1. A method of display, comprising steps of that a computing device inan image source which is separated from a display defines an area ofinterest in an image, and the computing device in the image source usesinformation pertaining to such area of interest to calculate anddetermine a System Synchronized Backlight Control (SSBC) profile of theimage and the computing device in the image source provides the SSBCprofile of the image pertaining to the area of interest in a videosignal stream or the computing device in the image source provides theSSBC profile of the image of the area of interest as metadata, whereinsaid SSBC profile means system synchronized brightness control which isa means of simultaneously adjusting backlight brightness and gray levelsof LCD pixels in real time based on the content of the input image. 2.The method of claim 1, said defining comprising defining the area ofinterest at the image receiver.
 3. The method of claim 1, said definingcomprising defining at the camera that senses the image.
 4. The methodof claim 1, said defining comprising defining the area of interest whileediting the image.
 5. The method of claim 1, said defining comprisingdefining the area of interest at a display for displaying the image. 6.The method of claim 1, said defining comprising defining the area ofinterest at an image player.
 7. The method of claim 1, said definingcomprising defining the area of interest ax a video recorder.
 8. Themethod of anyone of claim 1, wherein said defining is carried out by aperson selecting an area of interest.
 9. The method of anyone of claim1, wherein said defining is carried out automatically using analgorithm.
 10. The method of anyone of claim 1, said using informationcomprising providing for brightness adjustment.
 11. The method of anyoneof claim 1, said using information comprising providing for adjustmentof shades of gray (gray scale).
 12. The method of claim 11, saidproviding for adjustment of shades of gray comprising stretching therange of shades of gray of an input image to extend over a wider rangeof shades of gray,
 13. The method of claim 12, said stretchingcomprising interpolating.
 14. The method of claim 1, said providinginformation comprising encoding the image information pertaining to thearea of interest and providing the encoded information in the videosignal stream.
 15. The method of claim 1, said providing metadatacomprising providing the metadata in unused space of a video signalstream.
 16. The method of claim 1, said providing metadata comprisingproviding area of interest information in a video signal stream separatefrom the video imagery in the video signal stream.
 17. The method ofanyone of claim 1, further comprising a step that the computing deviceprovides information pertaining to such area of interest as metadata foruse in effecting adjustment of the image for display.
 18. The method ofclaim 1, wherein the SSBC profile is prepared where the image isobtained.
 19. The method of claim 1, wherein the SSBC profile isprepared where the image is prepared or provided for displaying. 20.Apparatus which is separated from a display for preparing a signal fordisplay, comprising means for determining an area of interest of animage, and a processor adapted to adjust an image signal based, on theimage characteristics of the area of interest, wherein said processorbeing adapted to compute for the image signal a System SynchronizedBacklight Control (SSBC) profile for the image based on the imagecharacteristics of the area of interest, said processor being adapted tocombine the SSBC profile as metadata with the image signal or saidprocessor being adapted to provide in an image signal stream the SSBCprofile of the area of interest and said processor being adapted toprovide the SSBC profile as encoded information in the image signalstream wherein said SSBC means system synchronized brightness controlwhich is a means of simultaneously adjusting backlight brightness andgray levels of LCD pixels in real time based on the content of the inputimage.
 21. The apparatus of claim 20, said processor comprising acomputer.